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Background 



In the spring of 1995, we were asked to review recent research on education and to 
summarize findings for busy policymakers. We had six weeks to research and prepare a 
summary briefing and an annotated bibliography on what works in schools and another 6 
weeks what doesn't work in schools. (The other two members of our project team 
conducted a parallel inquiry concerning workplaces.) In the course of completing this 
task, we identified findings that could be very useful to policymakers, educators and 
citizens seeking to improve their schools. That was the good news, and we conveyed it in 
our briefings and in our written report. 1 

The bad news is that empirically based information about what works in schools is so 
well hidden. We found no publication designed to summarize current knowledge on a 
range of school issues for the general reader. (The Department of Education's What 
Works is now over 10 years old.) We did find useful studies and scholarly review articles, 
but we are professionals accustomed to reading technical material who read specialized 
newspapers and journals, can get instant assistance from a librarian skilled in computer 
searches and from colleagues in the Department of Education and have ready access to 
the Department's library (now called the National Library of Education). People who 
lacked these specialized resources, we concluded after reflecting on our experiences, 
would be unlikely to come across this information either through original sources or in 
extracts or syntheses. 

A second piece of bad news is that there is so little solid information about what doesn't 
work. In the absence of such information, schools may continue to use and policymakers 
may even mandate practices that are unlikely to produce results. 

This paper captures our reflections on the utility of education research for policymakers 
and others interested in improving schools. We begin with observations from our 
experience about what policymakers want to know and about education and their 
preferences for how information is packaged. Next, we summarize our search for 
information and how we presented our findings. We then discuss several factors that 
limit the utility of educational research to general audiences currently, in the hope of 
stimulating discussion as to how they can be overcome. We close with some suggestions. 



'U.S. General Accounting Office, Schools and Workplaces: An Overview of Successful and 
Unsuccessful Practices (GAO/PEMD-95-28), August 1996. Information on how to obtain 
copies is presented in the appendix. 
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Meeting policymakers' information needs 



What policymakers want: focused answers to basic questions . 

In our project, congressional policymakers concerned with school improvement asked 
some very basic questions: "What works?" and "What doesn't work?" in elementary and 
secondary schools. They wanted information that would help them focus efforts and 
dollars on practices that are likely to produce results and avoid supporting or mandating 
practices that are ineffective. We think state and local policymakers and citizens have 
similar needs. 

Like others we have worked with, our policymakers preferred to get this information 
initially through a summary-oral or written in plain language-that focused on key findings 
relevant to their questions and could be absorbed in a short time (for example, in a 20- 
minute presentation or the first 2 pages of a written document). More detailed material 
that they could consult as follow-up to the initial summary was welcome. However, in 
the absence such a summary to guide them, policymakers find the typical detailed report 
difficult to digest. 2 

How these needs shaped our response 

We conducted a whirlwind search of recent literature summaries and reports that 
examined connections between school practices and student outcomes using valid and 
reliable empirical data. Publications that met our selection criteria were summarized in 
an annotated bibliography, using the format shown in Figure 1, page 3. We drew from 
these what we judged to be the most significant and consistent findings and highlighted 
them in oral briefings: one on what works and one on what doesn't work. 

We used the ten minutes of presentation time in each briefing to focus on a few key 
points, summarized in Figure 2, page 4 and presented in text form in the appendix to this 
paper. 3 (We combined the two briefings into one for purposes of written presentation, for 
reasons that will be discussed below.) Draft versions of the bibliography entries pertinent 
to each briefing were given to our audience at the time, to be drawn on for further 
information. A condensed version of the written presentation was included in our report, 
along with the bibliography entries themselves. 



2 For observations on providing policymakers with information they can use, see U.S. 
General Accounting Office, Program Evaluation: Improving the Flow of Information to 
the Congress (GAO/PEMD-95-1), January, 1995.. 

'Each briefing also included ten minutes on practices in the workplace, and half an hour 
or so for questions and discussion. 
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Figure 1: Annotated bibliography format 



Work Environment 

Lee and Smith 

Practice 

Setting 

Source of Information 
Findings and Conclusions 



Comments 



Valerie E. Lee and Julia B. Smith. “Effects of High School Restructuring 
and Size on Gains in Achievement and Engagement for Early Secondary’ 
School Students.” University of Wisconsin, Center for Organization and 
Restructuring of Schools, Madison, Wisconsin, April 1, 1994. 

Traditional organization, moderate reform practices, and “restructuring” 
or organizational practices that foster continuity in teacher-student 
contacts, cooperative teaching or learning, and staff involvement in 
common planning and school problem solving. 

Public and Catholic and other private high schools. 

Eighth- and lOth-grade test scores and other school and student data from 
the National Education Longitudinal Study of 1988. Analysis covers nearly 
12,000 students in 801 schools. 

Small size, academic emphasis, and restructuring are associated with 
greater student engagement and gains in learning. The distribution of 
engagement and learning gains across students of different social 
backgrounds is more equitable in restructured than in either moderately 
reformed or traditionally organized high schools, which are larger and 
more internally stratified than the other types and enroll higher 
proportions of low-income and minority students. 

Schools get best results when they build a strong core of operation by 
adopting only a few restructuring reforms— -along with such conventional 
reforms as emphasizing academic requirements, parent involvement, and 
staff stability. Small size makes both types of reform easier to implement. 



The scores used to measure growth in student achievement were not 
higldy reliable, and the survey data comprised relatively rough indicators 
of student engagement and school practice, fire authors cite case study 
research as evidence (hat. (heir findings rue on target despite these 
limitations. 



Figure 2: Topic headings for briefings 



Criteria of Success 


Student achievement at or above the expected level 

High teacher and student engagement in learning activities 

Effectiveness in overcoming disadvantage, such that students who are behind 
when they enter school are able to catch up 


Foundations for 
. Success 


Mission focused on learning with high expectations for all 
Organizational strategies to support the mission 



DIMENSION 


PRACTICES ASSOCIATED WITT 
SUCCESS 


PRACTICES NOT ASSOCIATED WITH 
SUCCESS 3 


Work 

Environment 


Safety, orderliness 


Large, fragmented secondary schools 




Sense of community 


Structural changes unconnected to learning 




Leadership that fits the situation 


Inattention to instructional improvement 




School-level problem solving 


Innovation overload 


Human Resources 


Control over entry and exit 


Individual merit pay 




Supportive induction 

Engagement in professional 
development 


Conventional approaches to teacher 
evaluation 

Conventional staff development 






Conventional recruitment, screening, and 
assignment of teachers 


Curriculum and 
Instruction 


Academically rigorous, focused 
curriculum 


Retention in grade and social prt notion 




Effective and engaging instruction 


Tracking students into low-level, remedial 
classes 




Chance for students to succeed 





Conclusions 


Much that is successful isn't new 




No single practice guarantees success, and various combinations can work 




Success is a matter of fit between mission and practice 



* Table entries for this column reflect practices whose effectiveness has at least been seriously questioned. 
Evidence that some of them don't work is fairly clear, and others are still the subject of scholarly debate. 



Barriers and limitations in the literature we reviewed 



Access 



As we conducted our search, it struck us that the ordinary person-relying on his or her 
nearby bookstore and library and on publications of general circulation-would be unlikely 
to come across most of the items we included in our bibliography. As shown in Figure 3, 
all but a few of the items appeared through specialized sources: social science 
publishers, professional journals, think tanks, and government offices. We checked the 
shelves of several chain book stores offering mostly popular selections and found none of 
the items or. our list. (These stores carried few books on any aspect of education, in 
notable cor trast to their selections on business.) The more extensive collection on 
education ir. more 'serious' bookstore included two of our items, both published by 
university presses. The local libraries we checked in Washington, DC and Fairfax County 
did not offer books from specialized sources. 

Even if original sources were difficult to unearth, policymakers and others interested in 
school improvement might gain access to their findings through other channels. For 
example, they might learn about successful and unsuccessful practices from articles that 
repackage findings from a particular study or synthesize the evidence from a body of 
work in educator-oriented journals or media of general circulation. We were disappon ted 
to find relatively little evidence of such repackaging of the work included in our report. A 
synthesis that evaluates and summarize findings from a body of studies in plain language 
can be a very effective in communicating to policymakers on big-picture issues. 

Readability 



Characteristics of research reports that decrease their readability poses additional barriers 
to policymakers' access to knowledge. Organization of the study report is one such 
barrier. In the material we reviewed, major findings and conclusions were often buried 
within the text or were presented at the very end of a study. Abstracts and introductory 
sections often described what authors did rather than what they found. Even diligent 
readers could be easily discouraged from completing a report when its usefulness was not 
quickly apparent. 

Technical terminology is also a barrier. Although it often simplifies communication 
among individuals engaged in similar research activities, it can block communication 
between researchers and policymakers. We noted that many studies, particularly articles 
in researcher journals, would not be comprehensible to readers who lacked training in 
research or statistics. 
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Figure 3: Type of publisher, name of publisher, and number of citations selected. 



Type of Publisher 


Name of Publisher 


Number 

Selected 


Social Science Publishers 


Cambridge University Press (1) 

The Falmer Press (1) 

Harvard University Press (1) 
Jossey-Bass (1) 

Longman (1) 

The New Press (1) 

Sage Publications (1) 

State University of New York Press (1) 
Teachers College Press (3) 

Technomic Publishing Company (1) 
Yale University Press (2) 


14 


Professional Journals 


American Education Research Journal (1) 

Early Childhood Research Quarterly (1) 
Education Leadership (2) 

Educational Evaluation and Policy Analysis (2) 
Journal of Educational Computing Research (1) 
Phi Delta Kappan (2) 

Review of Educational Research (2) 

The Urban Review (1) 


12 


Government Publications 


Office of Technology Assessment (2) 
US Department of Education (4) 

US Government Printing Office (2) 


8 


General Publishers 


HarperCollins (1) 

Basic Books (1) 

MacMillan Publishing Company (1) 


3 


Private Rese? .rch 
Organizations 


The Brookings Institution (1) 

The RAND Corporation (1) 

Center for the Organization and Restructuring of 
Schools (1) 


3 
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Finally, we thought that policymakers would have difficulty interpreting many of the data 
tables in studies we reviewed. This was especially so for tables that were used to 
summarize large amounts of statistical information rather than to present significant 
outcomes. When writing for policymakers, it may be worth the additional page space 
(and the forbearance of the editor) to imbed simplified graphics within the body of a 
report and place highly technical material within an appendix. 

The difficulty of saving what doesn't work 

We were struck by how difficult it was to summarize what doesn't work in schools. This 
difficulty stemmed from three main factors. First, the question of what doesn't work is 
often addressed only indirectly, if at all. Most of the material we found was oriented 
toward finding and highlighting what does work or on factors that contribute to school 
success rather than those that impede it. Such studies may imply that if X works, then its 
absence or its opposite doesn't work. But such an implication may well not be 
warranted. 

A number of the studies included in our bibliography contained information that shed 
light on what's ineffective, including case study descriptions of abominable practice in 
ineffective schools. However, the poor practices described-though newsworthy in the 
local community-axe not news in research terms and tended not to be highlighted in 
study reports. Also, few authors were willing to come right out and say, baldly, "This 
doesn't work." Unless it is strongly stated, the message that something is ineffective 
doesn't come across. 

The second factor is uncertainty about what we mean when we say that a practice 
doesn't work. There is, first of all, the question of what is "the practice." Many 
"practices" in education are defined either very broadly or in ideal terms and, in practice, 
take varied forms. Second, there is the term "doesn't work." Our experience uncovered 
a number of possible meanings for this term: 

(1) The practiced has proven difficult or impossible to implement as intended. 

(2) It has not succeeded in most of the places where it has been tried. 

(3) Overall, it is associated with minimal or negative effects. 

(4) It yields smaller positive effects than alternative practices. 

We use the case of merit pay for individual teachers-the practice most uniformly 
regarded as unsuccessful in the research literature-to illustrate the problems generated by 
these different meanings. Merit pay has been difficult to implement and has fallen out of 
use after a few years in almost all the places where it has been tried. In view of its rocky 
history, many would conclude that merit pay "doesn't work." But those who favor merit 
pay on policy grounds (over the years, the push for merit pay has come largely from 
legislators) would say instead that, it works in principle and simply hasn't had a fair trial, 
citing the few successful cases that, can be found. 
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Third, evidence that an educational practice doesn't work is rarely unequivocal, and 
conclusions concerning a given practice may change as it is viewed in a new context or 
as new information comes to light. Tracking and retention in grade are examples from 
our list. There is ample evidence that these practices as traditionally applied can be 
detrimental to the achievement of students that are in academic difficulty. But recent 
studies suggest that they can have legitimate and beneficial educational uses as well. 

Taken together, these factors make it difficult to discuss what doesn't work separately 
from what does. Hence, our decision to combine the two in our written presentation. 

Technical weaknesses that limit utility 

Many of the studies that we reviewed but did not select contained major limitations. 
Design constraints, contextual omissions, limited outcomes, and author biases wer more 
pervasive than we had anticipated. 

Single case studies were the basis for many reports. Few studies used longitudinal design 
or included adequate controls. We understand that all applied research designs have 
intrinsic limitations and that resource constraints may limit them further. However, 
greater efforts to strengthen research designs-perhaps through increased collaboration- 
could substantially increase the generalizability of much research. 

Omissions of adequate descriptions of demographic characteristics and contextual 
conditions were troublesome. After reading a study about what works, we easily 
imagined a policy maker reading the same material, asking whether the treatment would 
work for them and how, and not finding answers to these questions. 

Few studies reported findings in terms of student outcomes. Those studies that did often 
used limited outcome measures. For example, norm-referenced tests and SAT scores 
were used to measure classroom achievement. 

Meta-analyses can effectively summarize findings across large numbers of studies. 
Unfortunately, many of the meta-analyses that we reviewed offered little guidance l or 
improving practice. Descriptions of the original research and the criteria used to 
determine outcome selection were not provided, leaving the reader without a basic 
understanding of what the original treatments were, how original outcomes might have 
differed, how contextual variables might have influenced these outcomes, or whether the 
original research was sound. 

Finally, we found many policy pieces, supported by shaky assumptions and selective data, 
presented as objective research. We would not be surprised to find this genre growing in 
amount and sophistication as the internet explodes the accessibility of data. 
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Conclusions and suggestions 



We reviewed an extensive body of recently published reports, journal articles, and books 
about elementary and secondary school practices. Although almost all these works would 
have been useful to members of the research community, few offered accessible guidance 
to policymakers. The research community could do much to move policymakers from 
sound-bite to substance by synthesizing findings of quality research, rep; ickaging them in 
plain language and communicating them through channels that reach the general public. 

We hope that this paper will stimulate ideas as to how policy-relevant communication 
between the educational research community could be improved. Our suggestions are as 
follows. 

First, researchers themselves could take on the task. Repackaging rich and complex 
findings into a few simply presented points is difficult for someone trained as an 
academic. Still, research teams can creatively repackage their findings to reach wide 
audiences. For example, researchers at the Center on Organization and Restructuring of 
Schools recently summarized the results of a five-year research program in a "report to 
the public and educators" that is being distributed by the American Federation of 
Teachers, the Association for Supervision and Curriculum Development, the National 
Association of Elementary School Principals, and the National Association of Secondary 
School Principals. 4 

Second, researchers can enlist the aid of journalists. The educational desks of many 
newspapers are staffed by knowledgeable, interested individuals who will report on 
research studies if studies are timely, likely to be influential, and well-packaged. 
Journalists skilled at handling research material can also be found at trade-papers like 
Education Daily and Education Week which reach general audiences especially attuned to 
education. 

Finally, the internet offers a extraordinary powerful tool for making information 
accessible. We hope that individual scholars and the various institutions charged with 
getting information to the public will work together to make well-grounded, policy- 
relevant findings such as those we summarized easy for even the novice browser to find. 



4 Fred M. Newmann and Gary G. Wehlago, Successful School Restructuring: A Report to 
the Public and Educators by the Cent er on Organization and Restructuring of S< hools . 
Madison, WI; Center on Organization and Restructuring of Schools, 1995. 



APPENDIX 



SI 'CCESSFUL SCHOOL PRACTICES 

The criteria for school success that we drew from the literature were (1) student achievement at or above 
the expected level, (2) high teacher and student engagement in learning activities, and (3) effectiveness in 
overcoming disadvantage, such that students who are behind when they enter school are able to catch up. 
Schools that met all three criteria of success had a well-defined mission: they knew what they wanted to 
accomplish. Importantly, the mission was focused upon student learning and held high expectations for 
learning and behavior for all students, including those who needed extra assistance. 

Characteristics of the Work Environment 

Safety. Orderliness, and a Sense of Community 

Successful schools provided a safe and orderly work environment, in which instruction is protected against 
disruptions. In the case study literature, such schools were characterized by a sense of community, which 
was important for teacher and student engagement. The "community" included all students, not just the 
more able. Members of the community shared a sense of mission, as well as educational goals and 
experiences. Staff had close, sustained contact with each other and with students, who felt that teachers 
cared about them. Teachers collaborated in decisions about instruction. 

Practices that involved parents in their child's learning contributed to such an environment, as did adult or 
peer mentor programs that gave at-risk students a personal connection to school. Structural features such 
as small school size and scheduling and assignment practices that kept teachers with the same group of 
students for extended periods also made it easier to build a learning-centered environment. However, these 
were i >t necessarily successful practices by themselves. 

Instructional Leadership That Fits the Situation 

Leadership in framing a learning-oriented vision and focusing school efforts on improvement helped to 
create a productive work environment. We found that the source of leadership varied with the situation. 
Principal's leadership was typical of effective schools that serve low-income students-schools in which 
teachers tend to be relatively inexperienced. In other situations (including effective middle-class schools) 
experienced faculty took the lead on instruction and the principal pi-\ ed a managerial role. (Some teacher- 
designed alternative schools even did without a principal.) By contrast, case studies of ineffective schools 
found that no one took the lead on matters of instruction and learning; teachers and principal mistakenly 
believed that what they had traditionally done was good enough, and they did not seek to improve their 
practice. 

School-Level Problem-Solving Authority 

Finally, successful schools had the authority to solve problems at the local level, and involved members of 
the school community in this decisions. It is important that the problem solving activities focus on 
improving teaching and learning. Involvement of teachers, parents, and others in the school community is 
also important, but it can take a variety of forms. Studies of school site decision-making found that it was 
the fact and focus of involvement, rather than the specific means (such as a school council), that counted. 
Formal governance mechanisms like school councils have shown mixed results, in part because they treated 
governance as an end in itself rather than as a means to improving student learning. Many school-based 
management programs failed to provide participants with the skills and resources they need to use their 
powers effectively, did not focus on improving student outcomes, mid did not hold councils accountable for 
achieving improvements. 



BEST COPY AVAILABLE 
12 



APPENDIX 



Human Resource Practices 



Control Over Entry and Exit 



Studies have found that successful schools often exercised control over the entry and exit of staff and 
students. This control, which has enabled the schoo *s to build a stable staff and attract students who 
support its mission, has typically been associated with private schools and selective public schools (such as 
high schools of science). However, case studies have consistently demonstrated that principals and staffs 
in neighborhood public schools can influence entry and exit as well. 

Additionally, school-based hiring authority sometimes allowed the selection of i ichers with skills that 
match the needs of a particular site. Traditional centralized hiring and staffin- practices have not worked 
well in this regard. Several studies concluded that district officials, particularly in large urban systems, 
were often unwilling or unable to spend time to evaluate candidates in terms of their potential for specific 
positions. 

Control over teacher retention and attrition was as important as control over entry. According to several 
researchers, strategies that supported control in this area included allocating resources for multiple 
observations and evaluations by knowledgeable staff of new teachers, separating ineffective teachers from 
service, and providing separation initiatives such as cash bonuses to tenured, inadequately skilled staff. 

Support for Professional Development 

Since 1980, research evidence has demonstrated that professional development can enhance teacher 
performance and student learning. Effective approaches to staff development include 

- providing support for activities initiated by teachers to meet individual learning goals; 

- systematic observation and assessment of classroom teaching accompanied by feedback and 
discussion; 

- participation in curriculum development of school improvement processes; and 

- training that provides the presentation and demonstration of new skills, opportunities for simulated 
practice, and opportunities for coaching and practice in classroom situations. 

In contrast, traditional staff development employs an expert to provide training on a centrally selected topic 
to large numbers of teachers over the course of a school day with little or no follow-up activities in the 
class. In the studies we reviewed, such training was generally considered unlikely to led to substantial 
improvement in practice. The studies found that practice has not necessarily improved when new 
approaches or technology were introduced without training. 

In successful schools, new teachers profited from peer-coaching, mentoring, and collaborative activities 
with experienced staff. Placing new teachers in classrooms where they were most likely to do well, in 
contrast to the common practice of assigning new teachers to the most difficult classrooms, was obviously 
advantageous. 

Various teacher compensation systems have been implemented to improve teaching. However, the studies 
that we reviewed agreed that no particular approach to compensation has been associated with success. 
Teachers value primarily the intrinsic rewards that result from reaching students and helping them to leant, 
and efforts to use pay to improve individual perfonnance (particularly those that put teachers in 
competition with one another) have fared poorly. Approaches that linked compensation to skills and 
responsibilities appeared to have more promise. 
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APPENDIX 



Curriculum and Instruction 



Academically Rigorous. Focused Curriculum 

Schools that, offered an academicall’ igorous curriculum appeared to be more successful than those that 
offered a less challenging curriculum. Rigorous curricula sustained students' interests, built upon their 
strengths, and signified high expectations for achievement. 

A consistent theme that emerged from our review concerned the relationship between a focused curriculum 
and enhanced learning and engagement, particularly at the high school level. High school students were 
more likely to participate in learning that is focused around their occupational or academic plans. 

Integrating curricula in different subject areas through unified academic and occupational themes and the 
organization of academic instruction to support training for specific occupations enabled teachers to 
provide in-depth experience in areas of value to students. With’out a focus and expanded instructional time, 
the broad scope of the curriculum in elementary and secondary schools has tended to encourage superficial 
coverage of curriculum topics and to discourage more meaningful instruction. 

Effective and Engaging Instruction 

Active instructional techniques, including hands-on experimentation, experiential learning, student initiated 
pursuits, and cooperative, group activities improved engagement with learning activities for students at 
every level. These strategies were often found to be more successful than passive techniques, such as 
workbook exercises and lectures. Activities that related to students' life experiences and that integrated 
vocational and academic instruction were particularly helpful for disadvantaged students. 

Exposure for All 

Most successful schools offered a rigorous curriculum and engaging instruction to high achieving students. 
Schools that succeeded at overcoming disadvantage provided these to all student, including those who 
needed extra assistance. In contrast, school that provided low achieving students with low-level or 
remedial curricula (instruction that emphasized the acquisition of basic skills such as word recognition, 
computation, and functional writing through rote and memory) failed to engagement them or to encourage 
them to develop higher-order skills (such as organization, abstraction, deduction, or problem solving). 

Chance for All to Succeed 

For some students, liigh achievement is a distant goal. Successful learning practices reward them for 
positive efforts and for continuing progress toward that goal. Strategies such as cooperative learning, 
teacher and peer tutoring, frequent reinforcement, and extended time for learning helped low achievers, 
particularly at early grade levels, overcome difficulties and participate in a vigorous milieu. 

When these strategies were implemented appropriately and in a timely manner, students to progressed 
appropriately, and retention and social promotion decreased. Neither retention nor promotion, without the 
provision of specialized support, successfully closed academic deficits. Research findings indicate that 
retention, without extra support, was more harmful than social promotion because it substantially increased 
the probability of dropping out of school. 
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APPENDIX 



GAO PUBLICATIONS: ORDERING INFORMATION 

The first copy of each GAO report and testimony is free. 
Additional copies are $2 each. Orders should be sent to the 
following address, accompanied by a check or money order 
made out to the Superintendent of Documents, when 
necessary. Orders for 100 more copies to be mailed to a single 
address are discounted 25 percent. 

Orders by mail: 

U.S. General Accounting Office 
P.O. Box 6015 

Gaithersburg, MD 20884-6015 
or visit: 

Room 1100 

700 4th Street NW (corner of 4th and G Sts. NW) 

U.S. General Accounting Office 
Washington, DC 

Orders may also be placed by calling (202) 512-6000 

or by using fax number (301) 258-4066, or TDD (301) 413-0006. 

For information on how to access GAO reports on the 
INTERNET, send an e-mail message with "info" in the body to: 

info@www.gao.gov 
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